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Executive summary 


The project 


The Tutor Trust is a Manchester-based charity that aims to provide affordable small group and one to 
one tuition to schools. The Trust recruits university students and recent graduates, which enables it to 
provide tuition at a competitive rate. It predominantly aims to support schools in challenging 
communities and pupils who are looked-after or eligible for free school meals. 


This evaluation assessed the impact of the Tutor Trust on the English and mathematics attainment of 
95 pupils in Years 6 and 7. Participating students could receive up to 15 hours tuition whilst in Year 6 
and a further 10 hours tuition in Year 7. The evaluation also explored schools’ perceptions of the need 
for affordable tuition and their assessments of the quality of the service provided, in order test the 
feasibility of the approach and provide formative feedback to the Tutor Trust. 


The project was funded by the Education Endowment Foundation (EEF), the SHINE Trust and 
Manchester City Council. A separate evaluation of the impact of the Tutor Trust on the GCSE results 
of pupils in Years 10 and 11 is available on the EEF website. 


Key conclusions 


1. Due to the study’s design and problems recruiting schools to receive tuition or participate in 
the evaluation, this evaluation has not provided a secure estimate of the impact of the project 
on pupil outcomes. 

2. Participating pupils made slightly less progress in both English and mathematics than those 
in the matched comparison group. However, this finding was not statistically significant, 
meaning that it could have occurred by chance. 

3. Schools involved in the qualitative interviews were positive about the tuition, keen to work 
with the Tutor Trust again, and largely confident that the tuition was beneficial for their pupils. 
All teaching staff and senior leaders interviewed believed that there was a need for more 
affordable, high quality tuition. 


quality of tutors greatly influenced the impact of the programme. 

5. To ensure pupils derive maximum benefit, it is recommended that the Tutor Trust continues 
to put additional mechanisms in place to monitor tutor performance and conduct and that 
schools also implement such mechanisms. In addition, classroom teachers need to be 
involved in the planning and management of tuition so that it is not viewed as a ‘bolt-on’ and 


complements work in the classroom. 


: : Security rating awarded as part of 
Security rating the EEF peer review process 


Overall, the findings from this evaluation have very low security. The evaluation was set up as an 
efficacy trial, meaning that it aimed to test the approach under ideal conditions in a moderate number 
of schools. However, limitations of the evaluation design, and problems with the number of participants 
that the Tutor Trust were able to recruit to the programme and the evaluation, means that the results 
should be treated with caution. 


The evaluation used a quasi-experimental design, which enabled a comparison to be made between 
participating pupils and other pupils who were similar in terms of their demographic and socio-economic 
characteristics. A weakness of this design is that it does not take into account the unobservable 
characteristics of pupils, such as their motivation or the quality of the leadership in their schools. As a 
result, the findings are less secure than findings from randomised controlled trials. 


It had been hoped that at least 100 pupils would receive tuition in both maths and English, and provide 
data for the evaluation. However, data was only received from 82 pupils for English and 59 pupils for 
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maths. This substantially reduced the likelihood that the evaluation would be able to detect an effect 
with security. 


The qualitative fieldwork undertaken was based on the views of 13 senior leaders; 7 classroom 
teachers; 21 Year 6 pupils and 16 Year 7 pupils who had received tutoring; and 10 tutors. 


Results 


On average, participating pupils, including those eligible for free school meals, made marginally less 
progress than pupils in the comparison group. However, the differences in outcomes were not 
statistically significant, meaning that they could have occurred by chance. Overall, it is not clear from 
this evaluation whether the approach had an impact on student learning. A large majority of tuition was 
delivered in small groups rather than one to one, but the evaluation was not able to assess the relative 
impact of different group sizes. Only pupils who received some tuition in both primary and secondary 
were included in the analysis. 


In general, teachers in participating schools were positive about the tuition provided and a large majority 
of the senior leaders we interviewed were keen to work with the Tutor Trust again. Tuition was perceived 
to be most effective when tutors possessed strong pedagogical skills and subject knowledge, and were 
able to engage and interact successfully with pupils and target sessions appropriately. Other important 
points noted included: the value of involving English and mathematics teachers in planning sessions; 
the potential to use tutors as a familiar face across the transition from primary to secondary school. To 
increase the consistency of tutor quality more formal feedback from schools could be introduced. 


How much does it cost? 


At the time of the evaluation the Tutor Trust would charge primary or secondary schools between £18 
and £26 for an hour of tuition. Based on groups of three pupils receiving 25 tuition sessions, the total 
cost of the intervention is estimated at approximately £185 per pupil. 


In this project primary schools paid a flat fee of £1,000 for tuition irrespective of the number of pupils 
tutored and secondaries were offered tuition free of charge. 


Group Effect size Estimated To UL aL Ava eclilale) 
months’ progress 


All pupils -0.08 -1 month 


(English) 
All pupils -0.04 -1 month? 
(maths) 


FSM pupils -0.05 -1 month 
(English) 
FSM pupils -0.02 -1 month 
(maths) 


1 Since this report was published, the conversion from effect size into months of additional progress has been 
slightly revised. If these results were reported using the new conversion, all would be reported as O months of 
additional progress rather than -1, with the exception of the English result for all pupils, which would remain as -1 
months. See here for more details. 
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Introduction 


Intervention 


The Tutor Trust is a charity based in Manchester that aims to provide affordable small group and one 
to one tuition to schools. It was established in 2011 following a £180,000 grant from the Education 
Endowment Foundation. The Trust recruits university students and recent graduates, which enables it 
to provide tuition at a competitive rate. It predominantly aims to support schools in challenging 
communities and pupils who are looked-after or eligible for free school meals. 


The Tutor Trust rigorously screens tutor applicants to ensure that they have the appropriate mix of 
academic skills, teaching ability and interpersonal skills. It accepts around 30 per cent of applicants. It 
also delivers structured training to equip tutors with the ability to: decide on a long-term tuition plan for 
pupils; assess pupils’ grades/levels; plan a tuition lesson tailored to the needs of the pupil (this includes 
using the seven-point tuition plan structure); manage pupil behaviour; give praise and encouragement 
effectively; identify a pupil’s preferred learning style(s) and alter tuition accordingly; and teach English 
and mathematics appropriately. The Tutor Trust also ensures that tutors have valid Disclosure and 
Barring Service checks and organises the tuition sessions. 


In this project, funded through an additional £78,000 grant from the Education Endowment Foundation 
and co-funding from Manchester City Council and the SHINE Trust, Tutor Trust tutors were paid to 
deliver tuition to pupils in Year 6 and again in Year 7. The schools selected the pupils who received 
tuition in Year 6 and these pupils were all offered tuition again in Year 7. Some pupils only received 
tutoring in Year 6 and so are outside of the scope of this evaluation. Generally, pupils had different 
tutors in Year 7 than they did in Year 6. All tutoring took place on the premises of the participating 
schools. Schools chose either one to one or small group tuition. The tuition was designed to meet the 
needs of individual schools and pupils, hence the format and content of sessions varied (though tutors 
were provided with a model on which to broadly base tuition sessions). Year 6 pupils received tutoring 
between January and July 2013. Once in Year 7, these pupils received tuition again between October 
2013 and June 2014. 


Based on the data provided by schools (see Appendix 4 for the schools’ pupil data form), fifteen English 
tuition sessions were delivered per pupil at primary school and two to ten at secondary school, with an 
average of six sessions at secondary. The number of mathematics tuition sessions delivered per pupil 
ranged from ten to 13 at primary school and one to ten at secondary school, with an average of 12 
sessions at primary and five sessions at secondary. Some pupils received tuition on a weekly basis, 
others, particularly in secondary schools, received tuition more frequently (e.g. in one school, pupils 
received tuition every day for two weeks). Tuition sessions lasted approximately one hour and took 
place in school hours or during after school sessions. Where tuition was provided in school hours, pupils 
missed a lesson to receive tuition.” 


Background 


The rationale for establishing the tuition programme was based on the effectiveness of private tuition 
according to previous academic studies. The evidence synthesised within the Sutton Trust-EEF Toolkit? 
suggests that one to one tuition, where a student receives intensive tuition, generates five months of 


2 Across the primary and secondary schools, pupils missed a variety of lessons for their tuition. Each 
school adopted their own approach, some choosing to take pupils out of English or mathematics 
lessons, others opting for students to miss other subjects. In some cases the same lessons were missed 
each week, in others, the pattern varied. 


3 Available: https://educationendowmentfoundation.org.uk/resources/teaching-learning- 
toolkit/one-to-one-tuition/ 
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progress on average, and that small group tuition generates four months of progress on average. The 
evidence base related specifically to tuition delivered by university students is weaker. However, a small 
number of US studies have found that programmes led by college students can lead to improvements 
in student outcomes (Elbaum et al., 2000), suggesting that this is an area worthy of further evaluation 
in the UK. 


Transfer to secondary schools is a time of particular vulnerability for pupils from disadvantaged 
backgrounds. Pupils experience a significant dip in their learning as they transfer to secondary schools, 
especially pupils from disadvantaged backgrounds, which contributes to the widening gap in 
performance between those from more and less advantaged backgrounds (Evangelou et al., 2008; 
Evans et al., 2010). 


The EEF commissioned this independent evaluation to establish whether the Tutor Trust is achieving 
its aims, and to what extent the model is perceived to be sustainable, in relation to demand and delivery 
of low-cost tutoring in schools in Manchester. The Trust has expanded since its inception and, in the 
two years it has been operating, has delivered over 25,000 hours of tuition to more than 3,000 pupils 
across over 100 primary and secondary schools. It has 450 tutors working in all ten Greater Manchester 
local authorities. 


Our evaluation focuses on the effectiveness of providing tuition to pupils in Year 6 and again in Year 7. 
It has also provided formative information to the Tutor Trust. We offered suggestions for improvements 
to the tuition service following the process evaluation in the primary schools, and again following the 
process evaluation in the secondary schools. The Tutor Trust has taken steps to address our formative 
feedback at each stage. 


An efficacy study looking at the effect of the Tutor Trust’s Key Stage 4 tutoring on pupils’ GCSE English 
and mathematics results is available separately on the EEF website. 


Objectives 


The central object of the impact evaluation is to provide an outcome evaluation of tutees’ results in 
English and mathematics: 


e Do pupils who have received tutoring obtain higher results than those in the comparison group 
in Year 6 and Year 7? 

e Do pupils who receive tutoring in Year 6 and Year 7 outperform those who only receive tutoring 
in Year 6?4 


The process evaluation focuses on the model and its implementation, including the following: 


e Is the service fulfilling the identified need (i.e. lack of high quality affordable tutoring)? Do 
headteachers and school staff agree there is such a need? 

e Are the tutors ‘high quality’; are they offering high quality tutoring? 

e Are tutors delivering the tuition sessions in line with schools’ expectations? 

e Is the Trust meeting its recruitment targets? 


Project team 


NFER directed and led the process and impact evaluations. Helen Aston was Project Director. Emily 
Buchanan led the process evaluation and Jack Worth led the impact evaluation. Nick Bent, Director of 


4 This objective was not met due to small sample sizes. Please see section on ‘Analysis’ for further 
detail. 
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the Tutor Trust was responsible for engaging schools in the tuition, gaining schools’ consent to 
participate in the evaluation and collecting specified student data. 


Ethical review 


The evaluation was delivered in accordance with NFER’s Code of Practice. The Tutor Trust secured 
schools’ consent to participate in the evaluation (see Appendix 1). Schools were also provided with ‘opt 
out’ consent letters for parents. These set out the requirements of the evaluation and asked parents to 
inform the school if they were not willing for their child to participate. Gaining opt-out consent is sufficient 
to satisfy the requirements of the Data Protection Act® for processing personal data (see Appendix 2) 
Additionally, NFER sought and secured consent from all interview participants to collect and use their 
anonymised data (See Appendix 3). Given the age of the pupils, it was deemed appropriate for head 
teachers to act in loco parentis and to give their consent for pupils’ involvement in interviews. 


5 Data Protection Act (1998) http:/www.legislation.gov.uk/ukpga/1998/29/contents 
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Methodology 


Impact evaluation design 


The evaluation used a quasi-experimental design to evaluate the impact of tutoring on test scores. 
There was no randomisation at either the school or pupil level. This means that there are limitations 
(discussed further below) to the extent that we can interpret the measured differences in test scores as 
the causal impact of tutoring. Given the strong existing evidence base related to tuition (particularly one 
to one tuition) this design was judged to be appropriate. In essence, the evaluation was designed to 
enable the evaluation team to check whether the effects were broadly similar to those found in previous 
studies.. To have the best chance of secondary schools agreeing to provide tuition in Year 7, the Tutor 
Trust targeted the feeder primary schools of the secondary schools with which they had an established 
relationship. 


The comparison group was formed of pupils who were in the same cohort and the same secondary 
schools as tutored pupils, but who attended primary schools that did not receive Tutor Trust tuition. This 
ensured that the reason the comparison pupils were not tutored was because the Tutor Trust was not 
working with their school, rather than because pupils were not selected based on individual factors. 
This design means the comparison group is likely to be a more robust comparison than a design 
comparing tutored pupils with their primary school classmates. The primary school classmates of 
tutored pupils did not receive tuition because of characteristics/reasons that the teacher was aware of. 
These characteristics are likely to be associated with progress in test scores, meaning any comparison 
between tutored pupils and their primary classmates is likely to be biased. Had the untutored 
classmates been in schools where the tutor trust was working, they may have been selected for tutoring. 
They are therefore a better estimate of what might have happened to tutored pupils if they hadn’t been 
offered tutoring: in other words, the bias is likely to be lower. 


The comparison group pupils are weighted in the analysis to balance the background characteristics at 
baseline. We did this by giving more weight in the analysis to comparison pupils that are more similar 
to tutored pupils, and giving less weight in the analysis to comparison pupils that are less similar to 
tutored pupils. The measure of ‘more similar’ we used was the propensity score (the propensity score 
estimates the predicted probability of being in the tutored group). To estimate the propensity score we 
used a logistic regression® with pupils’ prior attainment, gender, month of birth, eligibility for free school 
meals (FSM), English as an additional language (EAL) and special educational needs (SEN) as 
predictor variables. We drew this data from the National Pupil Database (NPD).’ The weight for 
comparison pupils was calculated to be the odds ratio of the propensity score: the predicted probability 
of being tutored divided by the predicted probability of not being tutored. The intervention pupils had a 
weight of one in the analysis. We calculated weights separately for English and mathematics because 
the group of tutored pupils differed between the two subjects. As a result, the average characteristics 
of comparison pupils are much closer to the average characteristics of tutored pupils than they would 
be if we simply compared the two groups. 


5 Logistic regression measures the relationship between a set of ‘independent’ or ‘predictor’ variables 
and a binary ‘dependent variable’, which in this case is whether a pupil was tutored or in the comparison 
group. 

’ The National Pupil Database (NPD) contains detailed information about pupils in schools and colleges 
in England. The data includes test and exam results, prior attainment and progression at different Key 
Stages for pupils in the state sector, and attainment data for students in non-maintained special schools, 
sixth-form and further education colleges. The database also includes information about pupils’ 
characteristics, such as gender, ethnicity, first language, eligibility for free school meals, special 
educational needs (SEN), pupil absence and exclusions. 
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Unlike in a randomised controlled trial, we cannot be confident that unobserved background 
characteristics that are a factor in schools’ decisions to nominate pupils for tutoring (Such as a pupil 
being particularly motivated to learn), or the unobserved characteristics of the schools the pupils attend, 
are balanced between the intervention and the comparison group at baseline. This means that our 
estimates of the impact have to be interpreted carefully: any differences in test score could also be the 
result of underlying differences that were not taken into account in the matching because we had not 
measured those characteristics. 


The NFER team designed pupil data forms which were administered to schools by the Tutor Trust (an 
example is provided in Appendix 4). These forms asked schools to provide information on the tuition in 
their school, including pupil names and background characteristics, tuition subject, tutor-pupil ratio, 
number of tuition sessions and whether or not tutees missed lessons for their tuition. The analysis was 
based on the information provided in these forms, matched with additional pupil level data from the 
National Pupil Database. 


Eligibility 


The Tutor Trust sought to recruit primary schools in Manchester to receive the tutoring. While any 
primary school in Manchester was eligible to receive tuition and the sample was self-selecting at the 
school level, the Tutor Trust specifically sought to recruit feeder primary schools of the secondary 
schools in which it was already delivering tuition. A core part of the model was that schools had freedom 
to select whichever pupils they wanted to receive tuition. There were therefore no pupil level eligibility 
criteria for receiving tuition. We asked schools to identify why they had selected individual pupils to 
receive tutoring, from a list of options (See ‘Participants’ section in Results). The main reason for 
selection was underachievement relative to expected attainment at Key Stage 2, which is assessed at 
the end of Year 6. 


Consent 


The Tutor Trust secured schools’ consent to participate in the evaluation (see Appendix 1). Schools 
were also provided with ‘opt out’ consent letters for parents. These set out the requirements of the 
evaluation and asked parents to inform the school if they were not willing for their child to participate. 
Gaining opt-out consent is sufficient to satisfy the requirements of the Data Protection Act® for 
processing personal data (see Appendix 2) Additionally, NFER sought and secured consent from all 
interview participants to collect and use their anonymised data (See Appendix 3). Given the age of the 
pupils, it was deemed appropriate for head teachers to act in loco parentis and to give their consent for 
pupils’ involvement in interviews. 


Intervention 


Tutors recruited and trained by the Tutor Trust were paid to deliver tuition to pupils in Year 6 and again 
in Year 7.° Some pupils only received tutoring in Year 6, but they are outside of the scope of this 
evaluation. Generally, pupils had different tutors in Year 7 than they did in Year 6. 


All of the tutors involved in the programme had to pass a two-stage recruitment process, administered 
by the Tutor Trust. The process was designed to ensure that tutors had the appropriate mix of academic 
skills, teaching ability and interpersonal skills. The first round or recruitment consisted of a written, online 
application form modelled on the Teach First application form and designed in consultation with Teach 
First and Pricewaterhouse Coopers (PwC). There were also minimum qualification requirements: 
primary tutors had to have at least an ‘A’ in maths and English GCSE (or equivalent). Additionally, tutors 


8 Data Protection Act (1998) http:/Iwww.legislation.gov.uk/ukpga/1998/29/contents 
° The tutors were trained in either primary or secondary tuition, or both. There was no specific training 
for the Year 6 to Year 7 transition. 
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were asked to detail their previous experience working with children, to provide a personal statement 
explaining their motivation for working with The Tutor Trust, and to answer competency-based 
questions looking at humility, resilience, planning and organisation, and creativity. The second round of 
recruitment involved a short face-to-face interview with Tutor Trust staff to probe applicants’ motives, 
experience, dependability and communication and interpersonal skills. 


Tutors received a minimum of two-and-a-half days of structured and unpaid training before starting 
tutoring. The training sessions comprised: 


e A three-hour evening session on safeguarding, professionalism, why tuition matters and 
working with The Tutor Trust. The session was led by Tutor Trust staff. 

e A full-day session on lesson planning, learning styles, behaviour management, varying lessons, 
differentiation, creating a scheme of work for a tuition assignment and progress tracking across 
an assignment with the pupil. An educational consultant led this session. 

e A full-day session on teaching numeracy and literary to Year 5 and 6 pupils, delivered by 
educational consultants, who were commonly current or former teachers or senior leaders. The 
focus was on subject-specific tasks; curriculum understanding; common mistakes and 
misconceptions pupils may hold; teaching tools; assessing the level that pupils are working at; 
and selecting suitable tuition material. 


Tutors were also given access to additional CPD sessions; a range of Tutor Trust online resources; 
regular opportunities to interact informally and share problems and solutions with each other; and 
support from Tutor Trust staff as needed. All tutors were introduced to a seven-point tuition plan to 
structure tuition sessions (introduction, remember, model, try, apply, secure, reflect) during their 
training, but encouraged to deviate from it as appropriate. Hence, the nature and content of tuition 
sessions varied with the needs of individual schools and pupils. 


Year 6 pupils received tutoring between January and July 2013, and again between October 2013 and 
June 2014 once they moved into Year 7.1° All tutoring took place on the premises of the participating 
schools. Schools chose either one to one or small group tuition. Table 1 below sets out the proportion 
of pupils who received individual, small group, or large group tuition and were included in our analysis. 
Note that there is a high volume of missing data, where schools opted not to provide this information. 


il ratios 


Table 1: Tutor to pu 


Primary Secondary 
Tutor-pupil ratio Maths English Maths | English 
No information given 28% 77% 44% 5% 
1to1 5% - 7% 18% 
1to2 28% 11% 21% 20% 
1to3 5% 7% 28% 57% 
1to4 32% 5% - - 
1to9 1% - - - 


Based on the data provided by schools (see Appendix 4 for the schools’ pupil data form), fifteen English 
tuition sessions were delivered per pupil at primary school and two to ten at secondary school, with an 
average of six sessions at secondary. The number of mathematics tuition sessions delivered per pupil 


10 |t was intended that the tuition took place in secondary schools between September and December 
2013. 
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ranged from ten to 13 at primary school and one to ten at secondary school, with an average of 12 
sessions at primary and five sessions at secondary. Some pupils received tuition on a weekly basis, 
others, particularly in secondary schools, received tuition more frequently (e.g. in one school, pupils 
received tuition every day for two weeks). Tuition sessions lasted approximately one hour. 


Pupils who received tutoring were compared to other pupils in the same secondary school. We do not 
know what other initiatives the comparison pupils have been involved in. It may be the case that other 
pupils received other initiatives, in which case the analysis compares Tutor Trust tuition with what would 
otherwise have taken place. 


The process evaluation highlighted a number of fidelity issues. The delivery model was deliberately 
flexible and non-prescriptive but, in some cases, the tuition was not delivered in the autumn term (as 
intended), and not all of the secondary schools agreed to offer tuition in both English and mathematics, 
which meant that some pupils did not receive the intended tuition in Year 7. A further fidelity issue 
concerns the quality of tutors. The Tutor Trust programme is intended to provide high-quality tuition, but 
some schools from both years of the process evaluation have raised issues with tutor quality.41 More 
information is provided under the sub-heading ‘Fidelity’ later in the report. 


Outcomes 


Tutor Trust tutoring in English and mathematics is designed to increase the development of reading 
and mathematical ability. Two tests provided by GL Assessment were selected to measure these 
outcomes: Progress in English 127? and Progress in Maths 127°, respectively. The tests are appropriate 
for the age of tutored and comparison pupils and are aligned with the National Curriculum in England. 
The tests were taken by pupils from February to May, 2014. They were administered by staff at the 
secondary schools and marked and systematically checked by trained NFER markers. Schools were 
asked to deliver the tests under exam conditions. 


The main analysis used the raw score (number of items correct) as the outcome variable, rather than 
an age standardised score, which could introduce floor and ceiling effects. Progress in English 12 has 
a maximum mark of 58 and Progress in Maths has a maximum mark of 50. 


Sample size 


The number of pupils that received tutoring was determined by Tutor Trust, participating schools and 
the available grant funding. In the academic year 2012-13 the transition tutoring programme intended 
to offer tuition to around 250-300 Year 5 and Year 6 pupils across a number of primary schools in 
Manchester. Around 200 of these pupils were intended to receive further tutoring from the Tutor Trust 
in Year 7 in the academic year 2013-14. This evaluation focused solely on the Year 6 pupils who went 
on to receive tuition in Year 7. 


The evaluation protocol** made a conservative assumption that the number of intervention pupils and 
the number of comparison pupils would each be 100, clustered in 30 primary schools with an intra- 
cluster correlation of 0.15. The correlation between pre-test and post-test was assumed to be 0.75 and 
the level of confidence 95 per cent. We estimated that such a sample would give us a 35 per cent 
probability of detecting an effect size of 0.2 and an 88 per cent probability of detecting an effect size of 
0.4. The sample size calculations suggested that the design had a 76 per cent probability of detecting 


11 The Evaluation of the Tutor Trust programme in Secondary Schools (forthcoming) also raises tutor quality as a fidelity issue. 
? For more information please see: http://www.gl-assessment.co.uk/products/progress-english 

43 For more information please see: http://www.gl-assessment.co.uk/products/progress-maths 

4 http://educationendowmentfoundation.org.uk/uploads/pdf/Launch_-_Tutor_Trust_Primary.pdf 
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an effect size of 0.34, the average effect size found by previous research on small group tuition, so the 
study design seemed to be adequately powered.1° 


The comparison group was drawn from a much larger number of pupils than the intervention pupils, but 
because the comparison sample is weighted by the propensity score odds ratio, the weighted number 
of comparison pupils was necessarily similar to the number of pupils in the intervention group. 
Therefore, the determining factor for the statistical power the actual analysis has is the number of 
intervention pupils that are analysed. 


However, the number of tutored pupils tested was below 100. This was largely due to four of the 
secondary schools withdrawing from the evaluation, and to two of the secondary schools only providing 
tutoring and/or testing for either English or mathematics, not both.*® The result of a smaller sample size 
(plus a lower than expected correlation between pre-test and post-test and a lower than expected intra- 
cluster correlation (ICC) coefficient) is a larger minimum detectable effect size (MDES): the original 
design estimated a MDES of 0.36, whereas the actual analysis has a MDES of 0.51 for English?’ and 
a MDES of 0.54 for mathematics.*® See the participant flow diagram below for more details of the actual 
sample size. 


Analysis 


The analysis was conducted using a weighted linear regression model.’? A set of covariates are 
included in the model to explain outcome variance and increase precision: these variables were prior 
attainment (average points for Key Stage 1 reading and writing, and Key Stage 1 mathematics points); 
age in months at post-test; gender; eligibility for free school meals (FSM) in the last six years; English 
as an additional language (EAL); and special educational needs (SEN). Data for all background 
characteristics, except for gender and age at post-test which were collected from schools, were from 
the National Pupil Database (NPD). 


No information was collected on whether a pupil was selected for tutoring but did not go on to receive 
it. Therefore, the analysis was not done on an ‘intention to treat’ basis?°. Because the sample of 
comparison pupils was drawn from primary schools that the Tutor Trust had no relationship with, it could 
not include pupils that opted out of tutoring, or had not been selected for tutoring. The analysis, 
therefore, estimated the average treatment effect among those that received the intervention. 


16 Teaching and Learning Toolkit: 
https://educationendowmentfoundation.org.uk/resources/teaching-learning-toolkit/small- 
group-tuition/ 

16 The schools were free to decide whether they would offer both English and mathematics tuition, 
despite the free offer provided by the Tutor Trust. The secondary schools that withdrew from the 
evaluation did so on the grounds that they did not have time to provide information on the tuition 
received and to administer the required tests. 


1” Correlation between pre-test and post-test = 0.68, rather than 0.75; number of pupils = 82*2 = 164; 
ICC = 0.13; number of secondary schools = 13. 


18 Correlation between pre-test and post-test = 0.68, rather than 0.75; number of pupils = 59*2 = 118. 
ICC = 0.13. Number of secondary schools = 13. 


191 inear regression measures the relationship between a set of ‘independent’ variables (also known as 
covariates) and a continuous ‘dependent variable’, which in this case is a pupil’s test score. The 
regression model is weighted so that comparison pupils that are most similar to tutored pupils make the 
biggest contribution to the comparisons in the analysis. 

20 ‘Intention to treat’ means that the analysis is based on the initial intentions to provide tuition, rather 
than whether or not pupils actually received tuition. Our analysis was not done on an ‘intention to treat’ 
basis: if a pupil was intended to receive tuition, but for some reason did not, they would be excluded 
from our analysis. 
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The protocol specified that Year 5 optional test data would be collected from schools and used as the 
measure of prior attainment. As this test is closer to the beginning of the intervention, it would be a 
better covariate than Key Stage 1 points because it is likely to be more highly correlated with the post- 
test outcome. Both Year 5 optional tests and Key Stage 1 points are preferable to Key Stage 2 points, 
because the Key Stage 2 assessments were taken by pupils after the primary tuition had finished. 
However, as many primary schools were unwilling to give Year 5 optional test data to NFER, we have 
used Key Stage 1 points as the measure of prior attainment as this is available for all pupils from the 
National Pupil Database (NPD). This is a deviation from the published protocol. 


We had intended to explore whether pupils who received tutoring in Year 6 and Year 7 outperformed 
those who only received tutoring in Year 6. We were unable to do this level of analysis due to the smaller 
than anticipated size of our intervention and comparison groups, and the small number of post-test 
scores for pupils who received tuition in Year 6 but not in Year 7. 


The protocol also specified that a multilevel model would be used to account for clustering of pupils 
within primary schools. There were two deviations from protocol on this part of the analysis. Firstly, we 
decided to cluster by secondary school as the size of primary school clusters was very small in many 
cases. This was due to the wide range of schools that comparison pupils had come from, which was 
unknown at the outset of the project. Also, as the intervention was at pupil-level and the outcome 
measure was taken in the spring term of secondary school, clustering at secondary school was most 
appropriate. 


Secondly, the method for accounting for clustering was changed from a multilevel model to simply 
including school indicators as dummy variables. If clustering is not taken into consideration, the models 
may result in an overestimation of intervention effects. However, a ‘fixed effects’ modelling approach to 
taking account of the hierarchical nature of the data is preferable to multilevel modelling where the 
treatment is at pupil level; the way pupils are selected into schools is not well understood; and the 
background data is not very detailed (Clarke et al., 2010). All three of these conditions apply to this 
analysis, so a dummy variable for each of the 13 secondary schools is included in each regression 
model, with the dummy variable for the largest school excluded as the base. 


Along with the main analysis of the impact of English tutoring on reading attainment and of mathematics 
tutoring on mathematics attainment, we conducted the same analysis for the sub-group of pupils eligible 
for FSM. Sub-group analysis examines whether the interventions are particularly effective for FSM 
pupils. Exactly the same regression model as the main analysis was run on a sample restricted to just 
pupils eligible for FSM. 


Process evaluation methods 


We collected qualitative data from six of the primary schools and five of the secondary schools in which 
the Tutor Trust delivered tuition.24 We selected schools that had received the most hours of tuition 
(based on data provided by the Tutor Trust) as we felt that they would be better placed to provide rich 
data on the implementation of the tutoring and its perceived impact than those schools which had only 
received a small volume of tuition. We also sought to include a range of school types within the 
qualitative sample. Replacements were made where schools were not able to participate in the 
research?*. When selecting the secondary schools, we also avoided those schools participating in the 


21 Out of the eight primary schools and 13 secondary schools that were included in the impact 
evaluation. 

22 Two primary schools and 3 secondary schools who were initially approached to participate in the 
process evaluation did not feel able to participate, and were subsequently replaced. This was largely 
due to the schools not having time to accommodate the evaluation, or because schools were non- 
responsive to our initial request. However, one secondary school was replaced as they had not received 
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separate evaluation of the Tutor Trust secondary school programme. NFER researchers collected all 
of the data. The Tutor Trust was not involved in the selection process and was not told which schools 
participated. 


We spoke to a cross section of Year 6 pupils, staff and tutors in primary schools receiving the tuition, 
repeating this methodology in Year 7. We did not seek to re-interview the particular pupils we 
interviewed in Year 6 when they had progressed to Year 7 (i.e. we did not seek to track particular pupils 
as they progressed from primary to secondary school in the qualitative strand of the study). The data 
contained interviews with: 


e 13 members of senior management 

e seven classroom teachers 

e 21 Year 6 pupils and 16 Year 7 pupils who had received tutoring 
e 10 tutors. 


The interviews took place in the primary schools in the Summer term in 2013, and in the secondary 
schools in the Summer term 2014. This was either after or near to the end of the tutoring period. The 
interviews (largely carried out face-to-face) allowed us to explore all aspects of the tuition in depth, and 
to gain a school-, tutor- and pupil-level perception of the tutoring and its effectiveness. 


The interviews with senior managers and classroom teachers covered the following topics: 


e contextual data on the school and its intake 

e how the tuition was implemented in school (e.g. number, frequency, duration and length of 
sessions, reasons for offering tuition and how pupils were identified, tutor-pupil ratios) 

e whether tuition took place as originally planned 

e satisfaction with the quality of tutors and tuition 

e Satisfaction with the service provided by the Tutor Trust 

e suggested improvements 

e impacts on pupils 

e whether the tuition would be beneficial in other schools and other areas. 


Interviews with pupils covered: 


e their experience of tutoring (e.g. whether they enjoyed it, what they thought of the tutors and 
the sessions) 

e whether the tuition was beneficial or not (and the reasons behind their answers) 

e if the tuition could be improved. 


Interviews with tutors addressed: 


e motivations for becoming a tutor 

e the training and support received from the Tutor Trust 

e whether they felt any improvements could be made to the training and support 
e their experiences of tutoring 

e impact on their tutees. 


Subject to the permission of participants, interviews were recorded and interviewers took handwritten 
notes. Interviews were summarised for each question (including verbatim quotes). The data was 
analysed using thematic codes in relation to each question/section of the interview schedule. The 
qualitative fieldwork was relatively small in scale and the sample size was insufficient to analyse data 


tuition when the process evaluation interviews were due to take place and another withdrew as they 
opted out of the entire evaluation (e.g. both the process and impacts strands). . 
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by type of tuition delivered or received — such as one to one tuition versus small group tuition — which 
was not an aim of the evaluation. 
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Impact evaluation 


Timeline 
Activity Timescale 
Recruiting primary schools April 2012 to April 2013 
Tuition in primary schools January 2013 to July 2013 
Recruiting secondary schools April 2013 to April 2014 
Tuition in secondary schools October 2013 and June 2014 
Testing in secondary schools February to May 2014 


Participants 
Recruitment 


The Tutor Trust recruited primary schools in the run up to, and during, the academic year 2012/13. 
Manchester City Council, who part-funded the primary pilot suggested certain primary schools to the 
Tutor Trust that they thought would welcome a partnership with them. The Tutor Trust’s partner 
secondary schools from the 2011/12 academic year also recommended local partner primary schools 
to the Tutor Trust. The Tutor Trust recruited secondary schools in the run up to, and during, the 2013/14 
academic year. The Trust sought to recruit those secondary schools who were receiving the largest 
numbers of pupils who had been tutored in Year 6. The Tutor Trust recruited schools through: directly 
approaching schools via letter, email and telephone; word of mouth recommendations; repeat 
customers; and through Tutor Trust promotion at local authority-wide events. 


Participating schools 


The characteristics of the pupils in the participating primary and secondary schools are provided in 
Tables 2 and 3. They are compared with the average respective characteristics of all primary and 
secondary school pupils in England. The Tutor Trust primary and secondary schools have more pupils 
with FSM than the average school in England (62 per cent of primary schools and 88 per cent of 
secondary schools fall into the highest 20 per cent of schools for their FSM intake, compared to 14 per 
cent and 18 per cent of primary and secondary schools in England, respectively). They also have lower 
scores in national examinations (e.g. 57 per cent of Tutor Trust primary schools fall into the lowest 20 
per cent of schools for their Key Stage 2 average points score, compared to 20 per cent of all primary 
schools). 


Tutor Trust primary schools were lower performing and more disadvantaged on average than 
comparison schools, and compared to all English primary schools. The only difference between Tutor 
Trust primary schools and all English primary schools that was statistically significant was that schools 
are more disadvantaged. 
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Table 2: Characteristics of participating secondary school pupils 


oN a Ea Popen eae 
Lowest 20% 18% 8% 
2nd lowest 20% 20% 31% 
GCSE total point score Middle 20% 20% 46% 
(quintile) 

2nd highest 20% 20% 0% 
Highest 20% 22% 15% 

Lowest 20% 16% 0% 

2nd lowest 20% 24% 8% 

PIEF aug? Middle 20% 24% 0% 
2nd highest 20% 22% 31% 
Highest 20% 14% 62% 
Academy 51% 31% 
Community school 32% 31% 
School type Voluntary aided 10% 38% 
Voluntary controlled 1% 0% 

Other 6% 0% 

Rural 17% 8% 

Urban 
Urban 83% 92% 
Outstanding 23% 8% 
Good 49% 69% 
Ofsted rating Requires 

improvement 24% 15% 

Inadequate 4% 8% 

Number of secondary schools 3,389 13 


Note: totals may not sum to 100% due to rounding. Chi-squared test of significance, p-values: GCSE quintile = 0.07, FSM quintile 
= 0.00, school type = 0.07, urban/rural = 0.40, Ofsted rating = 0.34. 
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Table 3 Characteristics of participating primary school pupils 


3 Comparison Tutor Trust 
Variable eee, primary primary 
Yes aTole) [-} Yes aTote) f=} 
Lowest 20% 20% 21% 57% 
2nd lowest 20% 19% 30% 14% 
Key stage 2 
average points Middle 20% 20% 20% 14% 
(quintile) 
2nd highest 20% 19% 14% 14% 
Highest 20% 22% 15% 0% 
Lowest 20% 21% 5% 0% 
2nd lowest 20% 21% 5% 0% 
Free school meals . é _ ‘ ; 
(quintile) Middle 20% 21% 16% 0% 
2nd highest 20% 20% 16% 13% 
Highest 20% 18% 59% 88% 
Academy 9% 13% 13% 
Community school 56% 43% 38% 
School type Voluntary aided 20% 33% 38% 
Voluntary controlled 13% 11% 13% 
Other 2% 0% 0% 
Rural 33% 2% 0% 
Urban 
Urban 67% 98% 100% 
Outstanding 18% 25% 25% 
Good 62% 55% 63% 
Ofsted rating Requires 
improvement 19% 19% 13% 
Inadequate 1% 1% 0% 
Number of primary schools 17,641 132 8 


Note: totals may not sum to 100% due to rounding. 


Table 4 sets out the reasons given for selecting the Year 6 pupils who received mathematics and/or 
English tuition in Year 6 and then again in Year 7. Some of the data is missing, where schools chose 
not to provide it. This is particularly the case for English. The data we have indicates that 
underachievement relative to expected attainment at Key Stage 2, which is examined at the end of Year 
6, is the most commonly cited reason. Pupils being at risk of not achieving Level 4 was also a motivating 
factor in requesting tuition in most primary schools. This is likely due to accountability pressure to reach 
the floor standard of 60 per cent of pupils achieving at least Level 4 in reading, writing and mathematics. 
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However, eight and five pupils respectively were given tutoring in order to achieve the relatively high 
Levels 5 and 6 at Key Stage 2, so pushing the most able was a motivation for some. 


Table 4 Reasons for selection of Year 6 English and mathematics tutees 


No reason given 50 57 
Underachieving 15 17 
SATs 9 10 
English Sentence structure, writing styles 7 8 
To assist with achieving KS2 reading level 3 1 1 
To assist with achieving KS2 reading level 4 3 3 
Total 87 100 
No reason given 21 22 
Pupil is underachieving 13 14 
SATs 34 35 
To assist with achieving KS2 mathematics level 2 2 
3 
Mathematics To assist with achieving KS2 mathematics level 13 14 
4 
To assist with achieving KS2 mathematics level 8 8 
5 
To assist with achieving KS2 mathematics level 5 5 
6 
Total 96 100 
Total number of pupils tutored in one or both subjects 103 


Note: includes all pupils that were assessed for eligibility and does not match the sample analysed because some were missing 
other data. See participant flow diagram. 


Participant flow 


Figure 1 sets out the number of Tutor Trust and comparison pupils included in the analysis. A number 
of factors contributed to the lower than anticipated number of pupils included in the analysis, which we 
outline below. 


Firstly, a far larger number of the pupils in our dataset received tutoring in mathematics (n=207) than in 
English (n=121). This was largely due to the choices made by the primary schools and, in some cases, 
the secondary schools about which subject(s) their pupils should receive tutoring in. Demand for tutoring 
in mathematics was higher than demand for English tuition. We had anticipated a more even split in the 
numbers of pupils tutored in each subject in our evaluation design and were initially concerned that our 
analysis on the impact of English tuition would be limited by the low numbers. In fact, the imbalance 
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was reversed, because a higher proportion of math tutees than English tutees attended secondary 
schools that did not administer the maths test. 


Second, many of the pupils tutored at primary school were not tutored at secondary school, because 
they went to secondary schools that did not participate in tutoring. Due to the level of administrative 
burden involved, four of the secondary schools that received the highest numbers of tutored primary 
pupils chose not to participate in the evaluation and/or the tutoring. Further, two secondary schools only 
administered one of the two tests we used as our outcome measure: one chose not to administer PiE 
tests and one chose not to administer PiM tests. This was sometimes due to the preferences of the 
subject teachers in those schools: some felt that the test would not be beneficial and hence opted not 
to administer it. One school opted only to provide English tuition for their pupils, despite large numbers 
(approximately 40) having received tuition in mathematics in their primary school. Again, this was due 
to the preferences of that school. 


This school level non-participation lowered the number of tutored pupils included in our analysis 
considerably. We excluded 30 per cent (n=36) of pupils who had received tuition in English and 71 per 
cent (n=146) of those tutored in mathematics because of it. Therefore the number of pupils tutored in 
mathematics (n=59) in the final analysis was lower than the number of pupils tutored in English (n=82). 


The number of secondary schools that chose not to administer the PiE and PiM tests was also a crucial 
factor in the size of the comparison group being lower than anticipated. Six of the 13 secondary schools 
that participated in testing only agreed to test the tutored pupils, which explains the high number of 
comparison pupils whose school did not administer either test. A large proportion of comparison pupils 
(70 per cent for English and 67 per cent for mathematics) were excluded from our final analysis for this 
reason. Schools choosing to administer only one of the two tests, as noted above, also adversely 
affected the number of pupils in the mathematics comparison group. It should be noted that the large 
difference between the number of comparison pupils assessed for eligibility and the number of 
comparison pupils analysed is not a concern in terms of biased attrition because of the quasi- 
experimental design. The comparison group was constructed after the data collection including every 
pupil that had test outcome and background data. 
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Figure 1: Participant flow 


Tutor Trust pupils receiving tutoring in English 
Assessed for eligibility (n=121) 
Excluded (n=39: 32%) 


"School did not participate in testing (n=34: 28%) 
«School only administered PIM test (n=2: 2%) 
« Individual pupil missing post-test score?? (n=2: 
2%) 
= Outside common support*4 (n=1: 1%) 
Analysed (n=82: 68%) 
FSM subset (n=61) 


Tutor Trust pupils receiving tutoring in mathematics 
Assessed for eligibility (n=207) 
Excluded (n=148: 71%) 


«School did not participate in testing (nN=111: 54%) 
"School only administered PIE test (n=35: 17%) 
e Individual pupil missing post-test score (n=1: 
0.5%) 
« Missing KS1 or background data (n=1: 0.5%) 
Analysed (n=59: 29%) 
FSM subset (n=43) 


Comparison Group Pupils: E = n(English), M = n(mathematics) 
Assessed for eligibility (E=M=2,312) 
Excluded (E=1,619; M=1,695) 


School did not administer either test (E=M=1,141) 
Attended Tutor Trust primary (E=M=256) 

School only administered one test (E=90; M=72) 
No KS1 or background data (E=80;M=92) 
Outside common support (E=11; M=65) 


Analysed (E=693; M=617) 
FSM subset (E=396; M=370) 


23 This could be due to, for example, pupil absence on the day of testing. 


24 Comparison group pupils that had characteristics that were outside the range of characteristics of the 
intervention pupils (and vice versa) were excluded from the analysis because their characteristics 
differed too much to make robust comparisons. More formally, if a comparison group pupil is ‘outside 
common support’ it means their propensity score is outside the range of propensity scores in the 
intervention group, and vice versa. 
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Pupil characteristics 


Table 5 displays the average characteristics of pupils in the English tutoring group and the comparison 
group. The third column shows the difference between the intervention and comparison group averages 
and the fourth column indicates whether the difference is statistically significant at the five per cent level. 
Average prior attainment is significantly lower in the intervention group, and tutees are more likely to be 
eligible for free school meals and less likely to have English as a first language.” The fifth to eighth 
columns show the same for the intervention group and the comparison group weighted by propensity 
score. The significance test indicates that after weighting there are no significant differences between 
the tutored and comparison groups on average. 


Table 5: Demographic data of English intervention and comparison pupils (full sample and 
propensity score weighted sample) 


Propensity score weighted 


Full sample sample 
Background variable Inter Comp | Diff | Sig Inter | Comp | Diff Sig 
ae na L pons: }4go |aa3 jag |* 133 131 -02 Ns 
Female 54% | 58% A% ns 55% | 54% -1% Ns 
Pupil Premium 75% | 58% -17% | * 74% | 75% 0% Ns 
Special educational needs 22% | 17% -4% | ns 22% | 22% 0% Ns 
English as an additional language | 7% 23% 15% |* 7% 7% 0% Ns 
Age in months at post-test 145 145 0 ns 145 145 0 Ns 
Weighted number of pupils 83 704 82 83 
Unweighted number of pupils 83 704 82 693 
No. of secondary schools 10 10 


Note: Inter = mean in the intervention group. Comp = mean in the comparison group. Diff = difference in means between 
intervention and comparison group. Sig = statistical significance of difference in means at 95 per cent confidence level: * = p<0.05, 
ns = p>0.05. Weighting for each pupil in the full comparison group sample = 1. Differences may appear to not be correct because 
of rounding. 


Table 6 displays the average characteristics of pupils in the mathematics tutoring group and the 
comparison group. The table shows that mathematics tutees are more likely to be eligible for free school 
meals and less likely to have English as a first language.° Again, the fifth to eighth columns show that 
after weighting there are no significant differences between the tutored and comparison groups on 
average. 


Table 6: Demographic data of mathematics intervention and comparison pupils (full sample and 
propensity score weighted sample) 


2° Tjur’s R-squared statistic (the average difference in propensity score between the two groups) is 0.04. 
6 Tjur’s R-squared statistic (the average difference in propensity score between the two groups) is 0.02. 
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Propensity score weighted 


Full sample sample 
Background variable Inter Comp | Diff § Sig Inter Comp | Diff Sig 
Average Key Stage 1 points: | 14.2 | 15.0 0.8 ns 14.2 (14.2 0.0 ns 
mathematics 
Female 54% | 56% 2% ns 54% | 54% 0% ns 
Pupil Premium 73% | 56% -17% | * 73% | 74% 1% ns 
Special educational needs 19% | 15% -4% ns 19% | 19% 0% ns 
English as an additional language | 14% | 26% 13% * 14% | 11% -2% ns 
Age in months at post-test 145 145 0 ns 145 145 0 ns 
Weighted number of pupils 59 682 59 57 
Unweighted number of pupils 59 682 59 617 
No. of secondary schools 12 12 


Note: Inter = mean in the intervention group. Comp = mean in the comparison group. Diff = difference in means between 
intervention and comparison group. Sig = statistical significance of difference in means at 95 per cent confidence level: * = p<0.05, 
ns = p>0.05. Weighting for each pupil in the full comparison group sample = 1. Differences may appear to not be correct due to 
rounding. 


Numbers analysed 


We included 82 tutored pupils and 693 comparison pupils in the analysis of English tutoring; and 59 
tutored pupils and 617 comparison pupils in the analysis of mathematics tutoring. The analysis excluded 
pupils that did not have data for the relevant post-test and pupils that had missing NPD data (Key Stage 
1 and background characteristics). Some intervention and comparison group pupils were excluded from 
the analysis because their background characteristics were not sufficiently comparable (i.e. the 
propensity score of an intervention pupil lay outside the range of propensity scores in the comparison 
group, and vice versa). Figure 1 sets out the number of pupils excluded from the analysis for each of 
these reasons. 


Applying the propensity score weights to the comparison pupils means the weighted number of pupils 
for English and mathematics was 83 and 57, respectively. The comparison group pupils were drawn 
from eight secondary schools. However, only seven schools were included in each analysis as one 
school only participated in the English test and another only participated in the mathematics test. The 
intervention pupils were also drawn from eight secondary schools but an additional five schools only 
tested the intervention pupils in their school. 
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Outcomes and analysis 


Table 7 presents a summary of results from the impact analysis, summarising the number of pupils in 
the analysis and the estimated effect size with confidence intervals. 


Table 7: Summary of impact analysis results 


English tuition vs. comparison -0.08 (-0.33, +0.17) 82 693 
Mathematics tuition vs. comparison -0.04 (-0.33, +0.25) 59 617 
English tuition vs. comparison (FSM only) -0.05 (-0.37, +0.27) 61 396 


Mathematics tuition vs. comparison (FSM 


-0.02 (-0.41, -0.37) 43 370 
only) 


Analysis of English tutoring 
The results of the main impact analysis for English tutoring are displayed in Table 8. 


Table 8 Analysis of Progress in English raw score 


95 per cent 


Coefficient Sega confidence 
error 3 
interval 

Intervention -1.00 1.57 -4.09 — 2.08 
Average Key Stage 1 points: reading and writing 1.69 0.20 1.30 — 2.08 
Female 1.49 1.46 -1.37 — 4.35 
Pupil Premium (eligible for FSM in the past six 
years) -1.93 1.62 -5.10 —1.25 
Special educational needs -2.03 1.85 -5.66 — 1.60 
English as an additional language 6.05 2.88 0.40 — 11.69 
Age in months at post-test 0.24 0.19 -0.13 -0.61 
Intercept 21.66 5.88 10.13 — 33.19 


Note: Number of pupils: intervention = 82, comparison = 693. Standard deviation of outcome: intervention = 11.98, comparison 
= 12.36. ‘Average Key Stage 1 Points: reading and writing’ and ‘Age in months at post-test’ both centred using sample mean. 
The linear regression model also included secondary school fixed effects, which explained 13 per cent of the total variance. 


The primary analysis aimed to test whether intervention pupils made significantly more progress than 
the matched comparison pupils between Key Stage 1 reading and writing and the Progress in English 
test taken in Year 7. Table 8 shows that there is no significant difference between the progress made 
in the intervention and comparison groups. The standardised effect size calculated using Hedges’ g is 
-0.08 (CI -0.33-0.17). 
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We also carried out a sub-group analysis to test whether the intervention pupils made significantly more 
progress than the matched comparison pupils among pupils eligible for FSM in the past six years (the 
definition used to determine whether the school received extra funding through the Pupil Premium). We 
did this through: 


e restricting the sample to just those pupils eligible for FSM in the past six years. Table 9 sets out 
these results; and 

e including an interaction term in the main regression model, measuring the differential impact of the 
intervention on FSM pupils compared to non-FSM pupils. The results are shown in Appendix A5. 


Table 9 Sub-group analysis of Progress in English raw score 


SS) eclalerslae| 95% confidence 


eoetieient error interval 

Intervention -0.56 1.91 -4,31 -— 3.19 
Average Key Stage 1 points: reading and writing 1.70 0.23 1.25 -2.15 
Female 1.64 1.71 -1.72 —4.99 
Special educational needs -0.07 2.19 -4.36 — 4.21 
English as an additional language 4.74 3.21 -1.54 — 11.03 
Age in months at post-test 0.09 0.22 -0.35 — 0.52 
Intercept 15.88 14.73 -12.99 — 44.76 


Note: Number of pupils: intervention = 61, comparison = 396. Standard deviation of outcome: intervention = 11.35, comparison 
= 11.93. ‘Average Key Stage 1 Points: reading and writing’ and ‘Age in months at post-test’ both centred using sample mean. 
The linear regression model also included secondary school fixed effects, which explained 16 per cent of the total variance. 


The FSM sub-group analysis identified no significant differences in progress between pupils in the 
intervention and comparison groups that were eligible for FSM. The standardised effect size was -0.05 
(Cl -0.37 - 0.27). 

Analysis of mathematics tutoring 


Table 10 sets out the results of the main impact analysis for mathematics tutoring. 


Table 10 Analysis of Progress in Mathematics raw score 


Standard 95% confidence 


USUI error interval 
Intervention -0.43 1.58 -3.52 — 2.66 
Average Key Stage 1 points: mathematics 1.81 0.21 1.40 — 2.22 
Female 0.55 1.54 -2.46 — 3.57 
Pupil Premium (eligible for FSM in the past 6 years) | -1.55 1.72 -4.93 — 1.83 
Special educational needs -0.95 1.99 -4.86 — 2.95 
English as an additional language 1.11 2.44 -3.67 — 5.89 
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yeclalerlae| 95% confidence 


Coefficient : 
age) e interval 


Age in months at post-test 0.03 0.18 -0.33 — 0.38 


Intercept 23.16 2.06 19.12 — 27.20 


Note: Number of pupils: intervention = 59, comparison = 617. Standard deviation of outcome: intervention = 10.24, comparison 
= 10.72. ‘Average Key Stage 1 Points: mathematics’ and ‘Age in months at post-test’ both centred using sample mean. The linear 
regression model also included secondary school fixed effects, which explained 13 per cent of the total variance. 


The primary analysis aimed to test whether intervention pupils made significantly more progress than 
the matched comparison pupils between Key Stage 1 mathematics and the Progress in Mathematics 
test sat in Year 7. Table 9 shows that there is no significant difference between the progress made 
in the intervention and comparison groups. The standardised effect size calculated using Hedges’ g is 
-0.04 (CI -0.33 - 0.25). 


We also completed a sub-group analysis to test whether the intervention pupils made significantly more 
progress than the matched comparison pupils among a restricted sample of pupils eligible for FSM in 
the past six years. Table 11 sets out these results. We analysed the FSM sub-group using an interaction 
term as well and the results of that analysis are shown in Appendix 5. 


Table 11 Sub-group analysis of Progress in Mathematics raw score 


Standard 95% confidence 


pee error interval 

Intervention -0.24 1.96 -4.15 — 3.47 
Average Key Stage 1 points: mathematics 1.72 0.24 1.28 — 2.21 
Female 0.91 1.88 -2.08 — 5.18 
Special educational needs -0.13 2.33 -4.58 — 4.35 
English as an additional language 1.46 2.57 -3.56 — 6.30 
Age in months at post-test -0.03 0.21 -0.51 — 0.32 
Intercept 19.25 2.26 14.86 — 24.01 


Note: Number of pupils: intervention = 43, comparison = 370. Standard deviation of outcome: intervention = 9.07, comparison = 
10.43. ‘Average Key Stage 1 Points: mathematics’ and ‘Age in months at post-test’ both centred using sample mean. The linear 
regression model also included secondary school fixed effects, which explained 7 per cent of the total variance. 


The FSM sub-group analysis identified no significant differences in progress between pupils in the 
intervention and comparison groups that were eligible for FSM. The standardised effect size was -0.02 
(Cl -0.41 - 0.37). 


Primary schools paid a flat fee of £1,000 for tuition irrespective of the number of pupils tutored. This 
provided up to ten hours of tuition (and associated preparation) per pupil for the academic year 2012- 
13. Up to ten hours of tuition (and associated preparation) were provided free of charge for the 
secondary schools.’ 


27 At the time of the evaluation, the Tutor Trust would have usually charged schools £18 to £26 for an 
hour of tuition in primary or secondary schools. The charge would have depended on the ratio of tutors 
to pupils: 1:1 tuition was charged at £18 per hour; 1:2 at £20 per hour, and 1:3 at £26 per hour. 
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The cost of managing and delivering the tuition exceeded the fees brought in by the tutoring during the 
course of the evaluation, with the difference made up by grant income. The Tutor Trust expects to cover 
its costs from tuition fees alone in the academic year 2014-15. 


At the time of the evaluation the Tutor Trust would charge primary or secondary schools not 
participating in the evaluation project between £18 and £26 for an hour of tuition. Based on groups of 
three pupils receiving 25 tuition sessions, the total cost of the intervention is estimated at 
approximately £185 per pupil. 


Education Endowment Foundation 27 


Tutor Trust: Primary 


Process Evaluation 


Outcomes 


At both primary and secondary level, pupils, staff and tutors were positive about the benefits of the 
tuition. Note however, that the size of the qualitative sample is limited, and the findings should be 
regarded as indicative. 


Progress and attainment. In both primary and secondary schools, teaching staff held mixed views on 
whether the tuition had helped pupils to advance their learning in the tutored subjects. Very few teachers 
felt that the tuition would have a direct impact on Key Stage 2 results. When probed, most of the 
teaching staff we spoke to felt it would be difficult to attribute this impact to any one intervention. Where 
progress was noted, teachers referred to pupils talking about topics in class that they had covered in 
the tutoring and making connections between the two. All tutors felt that they observed an impact on 
their tutees’ learning. One Year 7 tutor did assessments at the start and end of the tutoring period and 
noted real gains in pupils’ spelling, punctuation and grammar. 


At primary and secondary level, of all the groups of interviewees, pupils were the most positive about 
the tuition’s impact on their learning. They said that tutors had helped them to understand particular 
topics or to know how to reach higher levels in their assessments. Some pupils directly attributed their 
successful KS2 or entrance exam results to the extra help they had received from tutors. Year 7 pupils 
also felt the tuition was helpful as it allowed concepts to be more clearly explained (more than once) 
and that they got more individual attention: ‘when you are in class the teachers didn’t get round to you 
very much because they need to get round to others, but when its one to one it’s easier’; ‘...we had a 
full 45 minute session and if you still didn’t get the hang of it then we will do like another 15 minutes 
next lesson before you go onto something else’. 


Self-confidence, self-esteem and attitude to learning. The qualitative data suggests that the biggest 
perceived impact of the tuition at primary level was on pupils’ self-esteem and confidence. A senior 
leader described the tutoring as ‘a big boost for the children’s confidence’. Interviewees reported that, 
for children from disadvantaged backgrounds in particular, improving confidence around learning can 
be an important precursor to improving their attainment. 


Most teachers, and all tutors and pupils, talked about pupils gaining confidence in their tutored subject, 
with some being more willing to put their hands up in class or to share learning with classmates. One 
primary teacher said: ‘the children come back and say, “oh | did this with my tutor”. What is really nice 
is a group of children who didn’t have a lot of confidence are actually coming into a class and actually 
sharing what they have been taught and helping other children. It is very positive’. Teachers had also 
observed pupils contributing more on subjects in which they had received tutoring: ‘Before they wouldn't 
put their hand up, they wouldn't get involved but if they have done it privately with their tutor, they then 
put their hand up and the involvement’s there’. Year 7 pupils (and their teachers) felt they had also 
grown in confidence in the classroom: ‘when you are in a group, you don’t want to ask, some people 
feel embarrassed’, and another pupil stated: ‘when we went back in class we felt we knew what we 
were doing’. 


Enjoyment: In the main, teachers, tutors and pupils told us that pupils had enjoyed the tuition (in both 
their primary and secondary school). This in turn impacted on pupils’ subject enjoyment and enjoyment 
of learning more generally. 


At primary level, interviewees from several schools reported that pupils wanted to come to school more 
on days when they had a tuition session scheduled. Pupils were also generally happy to stay on after 
school to receive tuition, with one pupil commenting: ‘Sometimes we had to stay after school but it's 
worth it because you learn, and it's good for your education’. Some ‘loved’ the tuition and both primary 
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and secondary pupils reported that tutors made their sessions fun (for example by letting them plan 
sessions, giving rewards or treats, and doing computer-based activities). At primary level, working 
outside the classroom environment was also considered ‘exciting’ and some pupils enjoyed the stretch 
provided by the tuition. It created an opportunity to learn different and harder things than they did in 
class, and offered freedom from having to work at a pace that met the needs of lower-achieving pupils. 
At both primary and secondary school, many pupils enjoyed the tuition as it provided an opportunity to 
ask questions without an audience. One Year 7 teacher noted: ‘they have really enjoyed it. It has been 
a really positive thing and it has been in line with what we have been working on in class, and it has 
shown’. 


Transition: The tuition was intended to support pupils’ transition by providing additional academic 
support at the time when pupils from disadvantaged backgrounds are particularly vulnerable to a dip in 
learning. Tuition was therefore delivered in Year 6 and then in the Autumn Term of pupils’ first year in 
secondary school (Year 7). None of the primary schools had purchased tuition with the aim of improving 
pupils’ transition from primary to secondary school. Some interviewees were surprised to learn that this 
was an intended outcome of the programme, despite the Tutor Trust making this explicit at the outset. 
Furthermore, few staff or tutors felt that the tuition would have an impact on pupils’ transition to 
secondary school. However, when prompted, some of these interviewees commented that the 
improvements seen in attitudes to learning and self-confidence would help pupils to successfully 
transition. Staff in secondary schools also felt that impacts on transition were limited, but recognised 
the potential for impact. It was largely felt that the tuition would have had more impact had it been 
delivered earlier in the academic year, if there had been some communication with the primary schools 
about the content of the tuition they had received, and if there had been continuity in the tutors: ‘It would 
be nice if tutors could carry on working with them for that transition period...that would be a good bridging 
between the two schools, helping them to settle in and having that familiar face’. Staff in two secondary 
schools also noted that the tuition had developed friendship groups and helped pupils to work together 
better with a wider range of their classmates. 


Primary pupils were far more positive about the impact of the tuition on transition. At least some of the 
pupils from all five of the primary schools felt that the tutoring had helped them to prepare for secondary 
school. Some pupils told us that they felt less shy about going to secondary school, while others were 
more confident that they now already knew some of the subject content that they would need post- 
transition. Year 7 pupils were unsure as to the extent to which the tuition had eased their transition 
(perhaps unsurprising given that the majority of Year 7 tuition took place in the spring, not the autumn 
term). 


Pupils’ aspirations: Impact on aspirations was mentioned by interviewees from three of the primary 
case-study schools. Each talked about the value of having a university student come into school to act 
as a positive role model and show pupils what they could achieve. One senior leader noted: ‘Children 
are more motivated and inspired by university students. It shows them this is what you can achieve... 
you can get to university. Low aspirations are a real issue and the school works hard to say “you can 
do that”. The [Tutor Trust tuition] contributes to that’. Pupils and staff from secondary schools also noted 
that the tutors being of a more similar age to pupils than their teachers helped to establish rapport and 
engage pupils. 


Behaviour: Almost invariably, interviewees did not report any positive impacts on pupils’ behaviour. In 
most cases, this was because the tutees’ behaviour was already good. One secondary teacher 
commented that as the pupils were used to tutoring from their primary school experience, they were 
perhaps more willing to engage in the tuition than they might have otherwise been. In a minority of 
schools, however, while most tuition sessions were well controlled, there were some issues with tutors’ 
ability to manage poor behaviour. As such, these schools felt that the tuition had had a negative impact 
on the behaviour of some of the children: ‘there was a good handful of children who had issues pretty 
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much every week’. In two secondary schools, pupils’ behaviour was said to be problematic in one or 
two tuition sessions as they were not being stretched appropriately by their tutor. 


Does tuition benefit all pupils equally? 


There was no clear message from interviewees about the extent to which the tuition impacted on all 
pupils equally. Rather, interviewees felt that the quality of the tutor and their ability to relate to the pupils 
and to teach them appropriately was the cause of variations in impact. Primary school teachers who felt 
that impact had varied according to pupil characteristics, cited one or more of the following groups of 
pupils as getting more out of the tuition: boys; self-motivated pupils; pupils with a positive attitude (rather 
than those who felt they were being forced to do the tuition because they were struggling); high 
achieving pupils; and pupils whose first language wasn’t English. Secondary teachers and tutors 
thought the following benefited most: shy pupils; those who were performing at lower level 4s (as 
opposed to higher levels 4s and 5s); pupils that usually get overlooked (e.g. they are not particularly 
high or low achievers and are generally quite motivated and well behaved): ‘a lot of the grey kids, who 
never get noticed, actually get their chance to shine’. 


In a minority of cases, the tuition was reported to have some unintended consequences or negative 
effects. For example, in one secondary school, a tutor consistently turning up late led one pupil to feel 
very negative about his tutoring session and the fact that he was missing his favourite lesson (drama) 
to do English tuition. The tutor’s lack of reliability also had negative consequences for a range of staff 
in school who were concerned with keeping the pupil engaged until the tutor turned up, welcoming him 
back into lessons if the tutor failed to show and liaising with the Tutor Trust to report the unreliability. 


Implementation 
Is the service attractive to schools and pupils? 


The Tutor Trust tuition is attractive to schools, based on the interview data collected. Most of the primary 
schools accepted the tuition as they needed additional academic support for (Some of) their pupils. The 
tuition offered personalised learning, which schools hoped would have a direct impact on attainment. 
However, the primary schools involved in the process evaluation did not consider the tuition as a way 
to ease transition to secondary school. The tuition appealed to the secondary schools too. In most 
cases the secondary schools were already using Tutor Trust tuition for other year groups. The vast 
majority of both primary and secondary schools planned to work with the Tutor Trust again. Almost all 
interviewees stated that they would recommend the Tutor Trust to staff in other schools and spoke very 
highly of their professionalism and responsiveness: ’Based on providers we’ve used, the Tutor Trust 
are the most organised and professional’. 


Is the service fulfilling an identified need (i.e. lack of high quality, affordable tutoring)? 


Interviewees felt strongly that there is a need for high quality, affordable tuition in Manchester. All 
interviewed senior leaders felt that this was the case. In some cases, teachers referred to the inability 
of parents in their school and other ‘non-leafy’ areas to afford any private tuition, which meant that their 
children were at a disadvantage. 


Most of the teachers we spoke to felt that the Tutor Trust was successfully meeting the need for high 
quality, affordable tuition. Given that the primary and secondary schools each received either ten or 
fifteen tutor sessions per pupil, either free of charge or for a one off fee of £1,000 most school staff felt 
that the tutoring represented good value for money. However in making these judgements, they were 
often considering the actual costs of tuition, beyond their free of charge/reduced costs entitlement (e.g. 
primary schools were considering whether or not to buy it next year, and secondary schools were often 
using Tutor Trust tutors for other year groups). Two secondary schools noted that they can provide the 
tuition at lower cost in house but they still planned to continue working with the Tutor Trust: ‘We’ve 
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recognised that this strategy works for our pupils at key points during their education and it is something 
that will continue to be part of our processes. 


Are tutors and tuition sessions high quality? 


School staff who we spoke to believed that the quality of tutors was generally high and that the quality 
of tutors greatly influenced the impact of the programme. For teaching staff, ‘high quality’ meant 
possessing the relevant pedagogical skills; being able to engage and interact successfully with pupils 
and target sessions appropriately; having good subject and curriculum knowledge; and being 
committed, reliable, flexible to changing needs and willing to work with a range of different pupils. The 
Tutor Trust advertises its tutors as high quality; indeed, each tutor is subject to a recruitment process, 
a training programme and ongoing support and quality assurance workshops. Generally, teachers and 
senior leaders felt that tutors possessed sufficient curriculum or assessment knowledge, but in some 
cases, there was room for improvement. Some interviewees did not feel able to comment on the quality 
of tutors teaching style or curriculum knowledge; if they did not work closely with the tutors, they were 
unable to make that assessment. Across the two years of the process evaluation?®, some concerns 
over tutor quality remain. In the minority2? of cases where tutors did not meet the expected quality 
standards, schools felt that this could have negative impacts on their tutees. 


Are tutors delivering the tuition sessions in line with schools’ expectations? 


The tuition sessions were generally delivered in line with school’s expectations. Schools were able to 
prescribe the frequency and the timing of the tuition, and the Tutor Trust responded appropriately to 
these requests. By the time of the secondary tuition, senior leaders praised the capacity and flexibility 
of the Tutor Trust in being able to meet the school’s needs, sometimes providing large numbers of tutors 
in intensive time periods. One secondary school noted that the tuition did not start as early as intended 
in the school year, which meant that they delivered the tuition in a more intensive time period than was 
initially planned. The school was positive about this change and did not see it as problematic. Another 
noted the same shift in timescales, which meant that it was harder to take pupils out of lessons. 


There were some deviations from expectations in the primary schools. In one school, tutors did not 
base their sessions on the children’s personalised targets, as requested, and tutor withdrawal or 
replacement meant that some groups were rearranged, or that some tutors weren’t in attendance at the 
school’s briefing meeting. Two primary schools also noted that the tutoring period was shorter than they 
had expected, but they did not consider this particularly problematic. 


In the Tutor Trust meeting its recruitment targets? 


The Tutor Trust met its recruitment targets for schools, and succeeded in working with more schools 
than intended. The Tutor Trust intended that 80 per cent of pupils who received tuition in their primary 
school would go on to receive tuition in their secondary school. The data we have gathered for the 
impact evaluation suggests that this target was met for English tuition, but not for mathematics. It should 
be noted however, that the data set we have been working with is partial and might not truly reflect the 
numbers of pupils who received tuition. As noted elsewhere in the report, the Tutor Trust were unable 
to deliver tuition to secondary schools until later in the Academic Year than was intended, reflecting the 
extensive time and liaison required to recruit and gain access to schools. 


Barriers to delivery 


The following barriers to delivery were evident across the primary and secondary schools: 


28 And also in the evaluation of the Tutor Trust secondary tuition. 


2°Two of the case-study primary schools and one secondary school raised concerns over tutor quality. 
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e Variations in tutor quality and reliability. Tutor absences or withdrawal, lateness and (in primary 
schools) concerns over tutor conduct or ability to manage behaviour were felt to have limited 
the impact of tuition. 

e At secondary level, in most cases, tuition was implemented later than intended. This limited its 
potential to have an impact on transition. 

e A lack of understanding of what was happening in the tuition sessions. This was caused by 
insufficient communication between classroom teachers and tutors, and was a more significant 
barrier in the secondary schools. It was evident that senior staff in schools who liaised directly 
with tutors and the Tutor Trust were not necessarily liaising with the classroom teachers, who 
were sometimes completely removed from any communication about the tuition. 

e One tutor noted that not being able to communicate directly with their school, and vice versa, 
meant that messages were ‘lost in translation’ or sometimes not passed on. They felt that this 
had a detrimental impact on their ability to tutor effectively. 


e Lack of feedback for tutors. Tutors would welcome more direct feedback on their performance 
and tutoring content from both schools and the Tutor Trust. Some called for more quality audits 
and opportunities to have their progress reviewed. 


Key factors for success 


Based on the findings from our small-scale qualitative fieldwork, the Tutor Trust tuition is likely to be 
most successful in bringing about an increase in attainment when: 


e Tutors are high quality, possessing the relevant pedagogical skills; subject knowledge; and 
personal attributes. Tutors need to be able to engage and interact successfully with pupils and 
target sessions appropriately; have good subject and curriculum knowledge; and be committed, 
reliable, and flexible to changing needs and willing to work with a range of different pupils. 
Where tutors are of poorer quality, this can have negative impacts on their tutees. 

e English and mathematics teachers are involved in the planning and management of tuition so 
that it is not viewed as an add-on. It is important that liaison with tutors and the Tutor Trust is 
devolved to staff who can ensure that the tuition is most appropriate. 

e School time is continually invested in working closely with tutors, monitoring progress and 
establishing feedback mechanisms. 

e There is continuity in the tutor-tutee relationship throughout the tutoring period. This helps 
pupils to form trusting and effective relationships with their tutor, supporting personalisation of 
learning and consolidation of learning week on week. 

e The model of tuition is tailored to best meet the school’s and tutee’s needs. At primary level, 
several interviewees felt that one to one sessions were more effective and efficient than small 
group sessions. This was because tutors found it easier to track pupil progress and pitch work 
appropriately. Two primary tutors stated that engaging the pupils in after-school sessions was 
harder than in the sessions they delivered during school time, as the pupils were tired and 
wanted to be heading home. In contrast, after-school sessions in one secondary school were 
thought to have worked particularly well. 

e Consideration is given to the ways in which schools group pupils (for example, grouping by 
ability can mean that some groups are ‘harder’ to manage than others, whilst mixed ability 
requires more differentiation). 

e Particularly at primary level, there is a ‘get to know you’ session for tutors and tutees so that 
trusting relationships can be established prior to formal tutoring. 


Fidelity 


The Tutor Trust model of transition tutoring is flexible by design and primary and secondary schools are 
free to decide how best to deploy the tutors with their pupils. At primary level, the schools were also 
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free to choose which pupils would receive tuition. The schools used a combination of pupil-tutor ratios 
(ranging from one to one to small group tuition). Some schools chose to deliver the sessions after 
school, others opted to deliver them in school time, either at regular time slots each week, or at varying 
time slots to avoid pupils consistently missing the same lesson. 


Despite the flexibility built into the tutoring, it was not always delivered as the Tutor Trust had intended. 
The ten sessions of secondary tuition were due to be delivered in the autumn term but, in the majority 
of schools, the tuition was not delivered until the spring or summer term. Two schools noted that this 
considerably reduced the value of the transition tutoring. This deviation was due to the schools taking 
some time to fully understand the offer from the Tutor Trust and to set up the internal logistics to ensure 
that the right pupils received tuition in their intended subjects, as well as winning the support of teaching 
staff. In addition, in some schools, the tuition was delivered in a very intensive time period (in one school 
tuition took place every day for ten school days). This was in order to accommodate the timescales of 
the evaluation and, although the schools were happy to take on this model of tuition (one believing it 
would actually be more beneficial for their pupils), it may not always have been the first choice for how 
to make best use of the free tuition offer. Finally, not all of the secondary schools in the impact and 
process evaluation delivered both English and mathematics tuition for pupils who were due to receive 
tuition in both subjects. This meant that some pupils did not receive the tuition intended at secondary 
school. 


A key issue with programme fidelity also relates to quality of tuition. Across the two years of the process 
evaluation®°, some concerns over tutor quality remain. Given that the quality of the tutor and tuition is 
considered key to leading to impact, it is imperative that continued attention is given to ensuring all 
tutors are high quality. 


Formative findings 
How could the programme be improved? 


e The Tutor Trust should continue to address variations in tutor quality to ensure that all tutors 
are up to the required standard of conduct and performance. Building in additional quality 
audits, or establishing more formal feedback or performance reviews would be welcomed by 
schools and tutors alike. 

e The Tutor Trust should continue to provide good practice guidance to schools, with practical 
steps for how schools can make the most out of tuition. This should encourage schools to: 

oO better use subject teacher time, with a particular focus on better connecting the tutor 
sessions with classroom activities and pupils’ areas for development 

oO setup systems to monitor the content and progress made in tuition sessions 

o host ‘start-up’ or ‘get to know you’ events or better utilise the initial tutor meeting to 
enable staff and pupils to meet tutors and begin to build relationships. 

e Schools and the Tutor Trust might seek continuity between tutors across the transition from 
primary to secondary school. 

e The Tutor Trust should enhance clarity around its transition aim to ensure that teachers 
maximise on this aim and help to improve transition to secondary school. 

e It would be valuable for the tuition in secondary school to take place in the autumn term during 
the transition period for tutees. 

e The Tutor Trust should encourage communication between feeder primary schools and 
secondary schools (specifically regarding the focus that was taken in primary schools and that 
which is planned for secondary school). 


30 And also in the evaluation of the tutor trust secondary tuition (forthcoming) 
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Control group activity 


There was a matched comparison group for this evaluation, (but no ‘active’ control group). Rather, we 
asked the Tutor Trust to refrain from delivering tuition to the classmates*! of the pupils who received 
tuition in Year 6, so that we could draw a comparison group from them. At primary level, some pupils 
did not like the fact that they had been targeted for tutoring, whilst some of their peers had not. This 
was not a problem for the pupils that we spoke to at secondary level. As the secondary schools 
effectively inherited pupils with tutoring attached, some schools expressed concern that other Year 7 
pupils might have benefitted more from tutoring than those to whom tutoring was being given. 


31 The comparison group did not have Tutor Trust tuition at primary or secondary level, thought we do 
not know whether tuition was delivered by another provider. 
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Conclusion 


Key conclusions 


1. Due to the study’s design and problems recruiting schools to receive tuition or participate in 
the evaluation, this evaluation has not provided a secure estimate of the impact of the project 
on pupil outcomes. 

2. Participating pupils made slightly less progress in both English and mathematics than those 
in the matched comparison group. However, this finding was not statistically significant, 
meaning that it could have occurred by chance. 

3. Schools involved in the qualitative interviews were positive about the tuition, Keen to work 
with the Tutor Trust again, and largely confident that the tuition was beneficial for their pupils. 
All teaching staff and senior leaders interviewed believed that there was a need for more 
affordable, high quality tuition. 
quality of tutors greatly influenced the impact of the programme. 

5. To ensure pupils derive maximum benefit, it is recommended that the Tutor Trust continues 
to put additional mechanisms in place to monitor tutor performance and conduct and that 
schools also implement such mechanisms. In addition, classroom teachers need to be 
involved in the planning and management of tuition so that it is not viewed as a ‘bolt-on’ and 
complements work in the classroom. 


Limitations 


There are two main limitations with the impact analysis, one a feature of the design and the other a 
problem with the implementation of the evaluation. Firstly, the evaluation used a quasi-experimental 
design to estimate the impact of the tutoring. The second main limitation is that the number of 
intervention pupils was lower than had been anticipated at the design stage. With only one limitation it 
may have been possible to generate an evaluation of moderate security. However, together the quality 
of the assessment was severely limited. 


In order to confidently attribute any effect directly to the study, we would have carried out a randomised 
controlled trial (RCT). As this was not possible, we could not confidently attribute the estimated effect 
wholly to the tutoring. In an RCT the control group provides an ideal estimate of what would have 
happened on average if the intervention group had not received the intervention. All the observed, and 
crucially the unobserved, characteristics of the two groups are likely to be balanced on average at the 
start of the trial, so when outcomes are measured, the differences can be attributed to the intervention. 
While propensity score matching balances the observed characteristics at baseline, it does not 
necessarily increase the likelihood that unobserved characteristics are balanced. If unobserved 
characteristics are a determining factor of whether the pupil gets tutoring and also correlate with test 
score outcome, then the impact estimates could be biased, and the extent of the bias is unknown. 
Interpreting the model relies on the assumption that everything that matters for pupil outcomes is 
balanced on average between the two groups at baseline. 


The research design estimated at the outset that there was a high probability of detecting an effect of 
0.34, which is the average size found in previous research on small group tuition. That estimate relied 
on the conservative assumption that 100 intervention pupils would be tested and analysed for both 
maths and English. The maximum number of tutored pupils that could have been tested was 121 for 
English and 207 for mathematics, which would have meant we were more likely to detect smaller 
effects. However, the analysis was based on (often retrospective) school-level data. Some schools 
provided very patchy data, despite repeated requests by the Tutor Trust for further detail. Further, the 
process evaluation revealed that some schools did not keep accurate records of their tuition sessions 
(e.g. number of sessions per pupil, the tutor-pupil ratio). It is therefore possible that the data provided 
by schools may not be an accurate record of the tuition that actually took place. Some schools also 
refused to allow their pupils to be tested and others only agreed to participate in one of the two tests. 
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This meant that the sample of intervention pupils that were tested and whose results could be analysed 
was below even the conservative assumption made in the protocol: 82 for English tutoring and 59 for 
mathematics tutoring. This small final sample size severely limited the ability to detect any effects of 
reasonable size with the conventional 95 per cent level of confidence: the chances of us detecting an 
effect size of 0.34 were 48% for English and 42% for mathematics. Therefore, it was expected that that 
none of the effect size estimates were statistically significant. 


In terms of the process evaluation, a larger qualitative sample size would have enabled us to be more 
confident about the robustness of the findings. This would have been particularly helpful in light of the 
disconnect between the impact and process evaluation findings. 


The Tutor Trust evaluation took place in a small number of secondary schools in Manchester. Tables 2 
and 3 show that the schools where tutoring was happening were not representative of schools in the 
country. In particular, the Tutor Trust primary schools did not have similar average characteristics to 
schools nationally, nor to primary schools that are feeder schools to the same secondary schools in the 
same area. The difference in characteristics suggests the findings are not generalisable to a wide set 
of schools. 


Interpretation 


It is not surprising that there was no significant difference between the performance of the tutored pupils 
and their untutored peers due to the small numbers on which the impact analysis is based. It is 
disappointing that the data set was smaller than predicted as it made detecting any effect highly unlikely. 
Should the analysis be run again on a larger data set where the tutoring was delivered as intended, the 
results might be more positive. 


The findings do not align with those of other research (synthesised in the Teaching and Learning Toolkit 
for example**) that has highlighted the impact that one to one or small group tuition can have. However, 
the confidence intervals suggest that the effect of the tuition could be in the range of that seen in similar 
research. Our process evaluation also highlighted a number of perceived impacts on soft outcomes, 
such as self-esteem and enjoyment of learning, and on pupils’ learning as a result of the transition 
tutoring. 


Our sense is that the Tutor Trust transition tutoring has potential to lead to improved achievement for 
pupils as they progress from primary to secondary schools, based on the positive findings from the 
qualitative strand of the evaluation. However, its potential success as a transition-focused intervention 
— delivered around the time of transition, when pupils from disadvantaged backgrounds are particularly 
vulnerable to a dip in learning - hinges on it being explicitly linked to transition (in terms of the timing 
and focus of the tuition) and, in some part, to its delivery in school. If the improvements set out in the 
process section are implemented, and tuition is delivered at the end of primary school and the beginning 
of secondary school, then the transition tutoring has a much better chance of having a statistically 
detectable effect. This requires ‘buy-in’ from both feeder primary schools and receiving secondary 
schools at the outset of the primary tuition. It was difficult for the Tutor Trust to get some of the 
secondary schools to fully embrace the tutoring for their pupils, and this lack of buy-in may have diluted 
the potential effect of tuition where it was not well integrated into school. This is in line with other 
research that provides evidence that when the right design and administrative structures are 
implemented, volunteer tutoring programmes can be effective in primary schools (Tepper Jacob, Smith, 
Willard and Rifkin, 2014). The Teaching and Learning Toolkit also states that teachers should monitor 
progress made in one to one tuition to ensure that the tutoring is beneficial.*° 


32 Available: http://educationendowmentfoundation.org.uk/toolkit/ 
38 Available: http://educationendowmentfoundation.org.uk/toolkit/ 
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Across this evaluation and that of the tuition in secondary schools, small issues with tutor quality remain. 
Although schools only judge a handful of the tutors to be poor quality, these tutors may be partly 
responsible for the limited impacts observed. The Tutor Trust needs to continue its efforts to ensure 
that only the best quality tutors make it through their recruitment and training programme. However, 
some of this focus might usefully be shifted to schools. Whilst hosting tutors, schools need to recognise 
that they too have a role to play in enabling the most effective tuition delivery. Schools could help to 
drive up tutor quality by liaising more closely with their tutors, and providing more instruction and 
feedback. Liaising with intended secondary schools or feeder primary schools is also important in the 
context of transition tutoring. If schools better understood the aim of the transition pilot, some statistically 
significant impact might have been realised. 


An evaluation of the impact of the Tutor Trust on outcomes at GCSE is available on the EEF website. 
Future research and publications 


Further research with a larger sample size is needed to be able to determine the effect of the tuition. 
The sample should be large enough to facilitate sub-group analysis, to enable exploration of the 
relationship between pupil attainment and type of tuition delivery (one to one versus small group). 
Ideally the research design would take the form of an RCT, to attribute any effect to the tuition. Drawing 
on the learning from this evaluation, securing the buy-in of receiving secondary schools to receive the 
tuition and participate in the evaluation at the same time as that of feeder primary school will be crucial. 
A cost effectiveness analysis would also be a beneficial addition to future research. 
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Appendix 1: Example consent letters to secure primary and 
secondary school participation 


Example letter for primary schools 
Dear [headteacher’s name], 


As you may be aware, the National Foundation for Educational Research (NFER) is conducting an 
evaluation of the tuition provided by the Tutor Trust to pupils in Year 6 of your school. 


NFER would appreciate your help with the evaluation. We would be very grateful if you would agree to: 


e complete and return a pupil data form in Excel for all Year 6 pupils who are receiving or 
have received tuition from the Tutor Trust in this academic year. Part 1 of the form asks for 
details about these pupils; the tutoring you intend them to receive; why you selected them 
to receive tutoring; and which secondary school they have a place at. It needs to be 
completed and securely returned to the Tutor Trust by 19" April. Part 2 of the form asks 
you to provide some further details about the tuition each pupil delivered. It needs to be 
completed and securely returned to the Tutor Trust by 7" July. 

e accommodate a half-day case study visit from a NFER researcher if you are selected as 
part of the sample for this strand of the research. The researcher would want to interview 
a small number of staff and pupils about their perceptions of the tutoring and its impact. 

e provide the Tutor Trust with Year 5 STA/QCA test data for all Year 6 pupils, together with 
their names dates of birth and UPNs. The Tutor Trust will contact you about this in the 
Summer Term. 


This data will be invaluable in helping us to measure the impact of the tutoring on pupils’ learning. 
Pupils’ data will be treated with the strictest confidence and according to the rules laid down by the Data 
Protection Act and NFER’s Code of Practice. 


Named data will be matched with the National Pupil Database and shared between NFER, the Tutor 
Trust and EEF for research purposes. No individual school or pupil will be identified in any report arising 
from the research. Some analysis will be shared with the SHINE Trust, to inform their evaluation. No 
named data will be shared with the SHINE Trust. 


| hope that this is permissible with you. If you require any further information, please contact XXXXXXX. 


Yours sincerely, 
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Example letter for secondary schools 
Dear [Headteacher], 


Some of your Year 7 pupils received Tutor Trust tuition in Year 6 (and may be receiving tuition in Year 
7 too). The National Foundation for Educational Research (NFER) is evaluating the impact of the tuition 
on the attainment of disadvantaged pupils for the Education Endowment Foundation (EEF). We very 
much hope that your school will participate in the evaluation. This would involve: 


e providing us with a list of your main feeder primary schools by 17 Sept. 

e providing us with the names, dates of birth and Unique Pupil Numbers of your Year 7 pupils 
on the Whole Cohort Form provided by 18" Oct. 

e completing a short pupil data form on any Year 6 pupils who are receiving tuition from the 
Tutor Trust in Year 7 by 20 Dec. The form asks about tuition subject(s), sessions and 
whether pupils missed any lessons to receive tuition. 

e administering Progress in Maths and Progress in English tests to your whole Year 7 cohort 
between 13" and 17" January 2014. NFER will provide you with the tests, instructions and 
SAEs for you to return the tests to us for marking. The two tests should take only around 
an hour each to administer to the whole group of pupils. The tests and feedback on your 
pupils’ performance will be provided for free. 

e potentially accommodating a half-day case-study visit from a NFER researcher in Spring 
2014. The researcher would want to interview a small number of staff and pupils about their 
perceptions of the tutoring and its impact. 


This data will be invaluable in helping NFER to measure the impact of the tutoring on pupils’ learning 
and helping us to make our tutoring offer as effective as possible. 


In the evaluation NFER will match named pupil data for your Year 7s to the National Pupil Database 
and data will be shared between NFER, the Tutor Trust and EEF for research purposes. NFER will not 
identify any individual school or pupil in any report arising from the research. Pupils’ test responses 
and the other data will be treated with the strictest confidence and according to the rules laid down by 
the Data Protection Act and NFER’s Code of Practice. 


| hope that you are able to contribute to this valuable evaluation. If you require and further information 
on the tuition please contact XXXXXxX]. If you require any further information about the evaluation, 
please contact XXXXXXX. 


Yours sincerely, 
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Appendix 2: Example opt-out consent letter for parents 


Dear parent, 
Help with research into the impact of tuition on children’s learning. 


As you may be aware, your child is receiving tuition from the Tutor Trust. The National Foundation for 
Educational Research (NFER) is conducting an evaluation of the tuition, to find out how effective it is. 


As part of the evaluation, NFER is collecting data about pupils who are receiving the tuition. This will 
involve: 


e A pupil data form which the school will complete. This asks for some data about pupils, 
including their name, date of birth, Unique Pupil Number (UPN) and their involvement in 
the tutoring. 

e ~=An interview from a CRB-checked NFER researcher, who will ask some pupils what they 
think of the tutoring and how it has helped them. 


e Ashort, one-off maths and English test which NFER may ask your child to take when they 
are in Year 7. 


All of the above data will be treated with the strictest confidence and according to the rules laid down 
by the Data Protection Act and NFER’s Code of Practice. 


NFER will match your child’s named data with the National Pupil Database (NPD) and share it with the 
Tutor Trust and EEF. They will not use your child’s name or the name of the school in any report arising 
from the research. 


Participation in this research is voluntary — children can choose not to take part, if they do not want to. 
This letter is to ensure that you are happy for your child to be involved in the evaluation and for 
the researchers to use their data to help us with this research. 


This research is confidential and no child will be named in any report of this work. 
What do | need to do now? 


If you do not want your child to take part in the research, please complete the slip at the end of this 
letter and return it to your school. If you are happy for your child to take part, you don’t need to do 
anything. 


If you would like more information, please contact XXXXXX. She will be happy to answer any queries 
you might have. 


Thank you for your understanding. 

Yours sincerely, 

[Headteacher] 

Evaluation of the Tutor Trust Tuition 

Parent opt-out form 

Please only complete this form if you would prefer your child not to participate in this research. 


Your Child!S: MAM 2255 be cwivcaeieds oc vacducanninedwnes eiduawadienan srend canedwiaeedreioniegs 
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Your name (please print)............ 0.0 ce cece cece eee eee ee eee eee ee eens eee eee eae eee ee 
SIQNEG. iets eee a ide aera etl 
Date 


Please ask your child to return this form to [insert your preferred name/contact here] at [insert your 
school name here]. 


Please return this slip to your school within 5 days if you do not want your child to take part. 
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Appendix 3: Example letter to secure participation in case- 
study visits and example wording used at the beginning of 
interviews with school staff and pupils 


A Example letter 
Dear XXXXX, 


As you are aware, the National Foundation for Educational Research (NFER) is conducting an 
evaluation of the maths and/or English tuition currently being delivered to some pupils at your school 
by the Tutor Trust. Thank you for all of your assistance in providing data to the Tutor Trust to inform the 
evaluation this far. 


| would be extremely grateful if you would be willing to assist us further by allowing an NFER researcher 
to visit your school. The purpose of the visit would be to speak to some staff and pupils who have been 
involved with the tuition to gather their perceptions of its impact. 


During the visit, the researcher would like to speak to: 


The headteacher or another member of the leadership team 
The class teacher of the children receiving the tuition 

A tutor (if present at the school during the visit) 

A group of two to five children who have received tuition. 


We would be looking to visit your school in May, June or July 2013. Each visit would take no more than 
half a day. 


Visiting your school would allow us to gain valuable information to inform tuition delivery in future so 
that it can benefit other young people. We will incorporate the information provided during the visit into 
our evaluation reports to the Education Endowment Foundation (EEF). These reports may be made 
publicly available. 


Please be assured that all the information collected during the visits will be treated in strict confidence 
and according to the rules laid down by the Data Protection Act and NFER’s Code of Practice. No 
school or individual will be identified in any report arising from the research. 


We will contact you shortly to discuss the visit. In the meantime, please do not hesitate to contact me 
on XXXXX if you wish to discuss any aspect of this work. 


Yours sincerely, 
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B Example wording used at the start of interview with key staff 


Thank you for agreeing to be interviewed and for your support for this evaluation. As you are aware, 
NFER is currently evaluating the effectiveness and impact of Tutor Trust tuition in primary schools, on 
behalf of the Education Endowment Foundation. 


As part of the evaluation we are carrying out case studies of schools that have received tuition from the 
Tutor Trust in this academic year. The purpose of this interview is to explore why you chose to work 
with the Tutor Trust, and your perceptions of the quality and impact of Tutor Trust tuition. 


The information that you provide will be treated entirely confidentially. We will not identify any individual 
or school in our evaluation outputs. 


The interview will take up to 60 minutes. | would like to record it, to supplement my notes. The recording 
will be deleted once | have taken notes from it. Are you happy for me to do this? 


Is there anything that you would like to ask me about the evaluation? 


Cc Example wording used at the start of interview with year 6 pupils 
My name is XXXX, 


I’m here today to ask you about the [name which pupils know the TT scheme by] that you’ve been 
getting from [name of tutor]. The people who run the tuition want to know how the tuition has helped 
you, and how to make it better in future. 


This isn’t a test, and there are no right or wrong answers to the questions | will ask you. | just want to 
hear what you have to say. No one will know what you tell me today, other than me. | won’t write your 
name down next to anything. But, | will use what you say to help me to write a report for the people who 
run the tuition. | won’t put your name anywhere in the report and your tutor won’t Know what you said 
about them. Is that all ok? 


Are you happy for me to ask you some questions then? 


[If yes] Is to ok if | record our chat? | will delete it once I’ve written up my notes but | want to make sure 
| don’t forget anything you say. If you want to stop talking to me at any time, please just tell me, because 
that is absolutely fine and you can go back to your classroom. Is there anything you would like to ask 
me before we start? 


[If no] OK. 
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Tutor Trust Evaluation: Year 7 Pupil Data Form 
Academic Year: 2013-14 


Name of School 
School postcode 


DfE number 


Please use this form to record data on all Year 7 pupils who received Tutor Trust tuition. 
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Appendix 5: Regression results from sub-group analysis 


Table A1: Sub-group analysis of Progress in English raw score 


Standard 95% confidence 


Coefficient 


error interval 

Intervention -1.97 2.71 -7,.29 —3.35 
Intervention * Pupil Premium 1.34 3.04 -4.62 —7.29 
Average Key Stage 1 points: reading and writing 1.69 0.20 1.30 — 2.08 
Female 1.47 1.46 -1.39 — 4.32 
Pupil Premium -2.56 2.17 -6.81 — 1.68 
Special educational needs -1.93 1.87 -5.59 -1.72 
English as an additional language 6.05 2.81 0.55 — 11.56 
Age in months at post-test 0.24 0.19 -0.12 — 0.60 
Intercept 22.07 3.72 14.79 — 29.35 


Note: Number of pupils: intervention = 82, comparison = 693. Standard deviation of outcome (FSM pupils): intervention = 11.35, 
comparison = 11.93. ‘Average Key Stage 1 Points: reading and writing’ and ‘Age in months at post-test’ both centred using sample 
mean. The linear regression model also included secondary school fixed effects, which explained 12 per cent of the total variance. 


Table A2: Sub-group analysis of Progress in Maths raw score 


Standard 95% confidence 


eortiicient error interval 
Intervention -1.31 2.80 -6.79 — 4.17 
Intervention * Pupil Premium -0.20 3.25 -6.56 — 6.17 
Average Key Stage 1 points: mathematics 1.78 0.21 1.36 — 2.20 
Female 0.35 1.58 -2.74 —3.44 
Pupil Premium -2.21 2.32 -6.75 — 2.33 
Special educational needs -1.14 2.06 -5.19 — 2.90 
English as an additional language 2.39 2.44 -2.39 —7.17 
Age in months at post-test -0.11 0.18 -0.45 - 0.24 
Intercept 22.63 2.39 17.95 — 27.31 


Note: Number of pupils: intervention = 59, comparison = 617. Standard deviation of outcome (FSM pupils): intervention = 9.07, 
comparison = 10.43. ‘Average Key Stage 1 Points: mathematics’ and ‘Age in months at post-test’ both centred using sample 
mean. The linear regression model also included secondary school fixed effects, which explained 7 per cent of the total variance. 
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Appendix 6: Security classification of findings 


2. Power: 5. Threats to 
ke ign: Ee 3. Attrition: 4. : Sa ae 
Design What is ri ion Balance validity: 

What is the What is the Adjustment vatlalty: 
realialiaale lan) Adjustment to 


ES7-) le) me | ge) to rating to 
fo [=}¢-1e1 ¢-] 0) (=) P & account for issues 


quality of the 


design of the folUi mice)aemaal- account for 3 
fi ation? Sk evaluation? nate ? seh 
Uatl Vaiu : f 
start? ; ec interpretation? 


Evaluation design Implementation Analysis and interpretation 


No threats to validity 


Fair and clear experimental 
design (RCT) 

Fair and clear experimental 
design (RCT, RDD) 
Well-matched comparison 
(quasi-experiment) 
Comparison group with 
poor or no matching 


The final security rating for this trial is 0 @ . This means that findings from this trial have very low security. 


iabalaniced: orl Significant threats 


observables 


The evaluation was designed as a matched trial and could have achieved a maximum of 3 @ . Many variables were 
used to match intervention pupils with comparison pupils, including SEN and EAL status. However, the within- 
school selection of pupils is not well understood and indeed the intervention did not apply selection criteria at either 
the school or pupil level. Based on data provide by schools, for English 57% of pupils were selected for tutoring for 
‘other’ reasons, and 22% of pupils selected for maths were selected for ‘other’ reasons. Therefore, security of the 
design was reduced by one padlock. The evaluation was designed to detect a moderate effect size of around 0.35. 
Unfortunately, despite the best efforts of the Tutor Trust to engage schools to participate in both the tutoring and 
the tests, there was substantial attrition from both. Only 29% of pupils who received mathematics tuition and 68% 
of those who received English tuition were tested (i.e. overall 43% of the pupils in the intervention arm were tested). 
The propensity score matching method achieved balance between the groups at baseline, and there were no 
threats to validity. Therefore, the security of the conclusions was reduced to0 @. 
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Appendix 5: Cost rating 


Cost rating Description 


£ Very low: less than £80 per pupil per year. 
££ Low: up to about £200 per pupil per year. 


EEE Moderate: up to about £700 per pupil per 
year. 


ELEEE High: up to £1,200 per pupil per year. 


ELEELEE Very high: over £1,200 per pupil per year. 
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